> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.

# CLIP Overview

> Understanding the CLIP architecture and how it learns visual-semantic representations

# CLIP Overview

CLIP (Contrastive Language-Image Pre-training) is a neural network trained on image-text pairs that learns visual concepts from natural language supervision. OpenCLIP is an open-source implementation that enables training and using CLIP models at scale.

## Architecture

CLIP uses a **dual encoder architecture** consisting of two separate towers:

1. **Vision Encoder** - Processes images into fixed-dimensional feature vectors
2. **Text Encoder** - Processes text into the same dimensional feature space

<Frame>
  <img src="https://raw.githubusercontent.com/mlfoundations/open_clip/main/docs/CLIP.png" alt="CLIP Architecture Diagram" />
</Frame>

Both encoders project their inputs into a shared embedding space where semantically similar images and text are close together.

### Vision Encoder

The vision encoder transforms images into feature embeddings. OpenCLIP supports multiple vision architectures:

* **Vision Transformer (ViT)** - Default architecture, divides image into patches
* **ResNet variants** - Convolutional neural networks (ModifiedResNet)
* **ConvNeXt** - Modern convolutional architectures
* **Custom architectures** - Via timm integration

From `src/open_clip/model.py:133-206`:

```python theme={null}
def _build_vision_tower(
        embed_dim: int,
        vision_cfg: CLIPVisionCfg,
        quick_gelu: bool = False,
        cast_dtype: Optional[torch.dtype] = None
):
    # Vision encoder can be ViT, ResNet, or timm model
    if vision_cfg.timm_model_name:
        visual = TimmModel(...)
    elif isinstance(vision_cfg.layers, (tuple, list)):
        visual = ModifiedResNet(...)  # ResNet architecture
    else:
        visual = VisionTransformer(...)  # ViT architecture
    return visual
```

### Text Encoder

The text encoder processes text descriptions into embeddings. It uses a **Transformer architecture** with:

* Token embeddings
* Positional embeddings
* Multi-head self-attention layers
* Feed-forward networks
* Optional HuggingFace models (RoBERTa, etc.)

From `src/open_clip/model.py:209-262`:

```python theme={null}
def _build_text_tower(
        embed_dim: int,
        text_cfg: CLIPTextCfg,
        quick_gelu: bool = False,
        cast_dtype: Optional[torch.dtype] = None,
):
    if text_cfg.hf_model_name:
        text = HFTextEncoder(...)  # HuggingFace models
    else:
        text = TextTransformer(...)  # Native transformer
    return text
```

## Contrastive Pre-training Objective

CLIP is trained using a **contrastive learning objective** that aligns image and text representations:

1. **Batch Construction** - Sample N (image, text) pairs
2. **Encoding** - Pass images through vision encoder, texts through text encoder
3. **Similarity Matrix** - Compute N×N cosine similarities between all image-text pairs
4. **Contrastive Loss** - Maximize similarity for correct pairs, minimize for incorrect pairs

The model learns to:

* Push matching image-text pairs **closer** together in embedding space
* Push non-matching pairs **further apart**

See [Contrastive Learning](/concepts/contrastive-learning) for detailed loss function explanation.

## Image-Text Similarity Scoring

Once trained, CLIP computes similarity between any image and text through:

### 1. Encode Both Modalities

```python theme={null}
# From README example
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    # Normalize to unit vectors
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
```

### 2. Compute Cosine Similarity

From `src/open_clip/model.py:347-354`:

```python theme={null}
def get_logits(self, image, text):
    image_features = self.encode_image(image, normalize=True)
    text_features = self.encode_text(text, normalize=True)
    
    # Scaled dot product (cosine similarity with temperature)
    image_logits = self.logit_scale.exp() * image_features @ text_features.T
    
    if self.logit_bias is not None:
        image_logits += self.logit_bias
    return image_logits, text_logits
```

### 3. Temperature Scaling

The `logit_scale` parameter controls the sharpness of the similarity distribution:

* **Higher temperature** (lower scale) → softer, more uniform probabilities
* **Lower temperature** (higher scale) → sharper, more peaked probabilities

Initialized as: `np.log(1 / 0.07)` ≈ 2.66 (learned during training)

## Key Properties

### Joint Embedding Space

Both vision and text encoders project into the **same dimensional space** (typically 512 or 768 dimensions), enabling direct comparison.

### Zero-Shot Capability

CLIP can classify images into categories it wasn't explicitly trained on by comparing image embeddings with text embeddings of class names. See [Zero-Shot Classification](/concepts/zero-shot-classification).

### Flexibility

CLIP can be used for:

* Image classification (zero-shot or with prompts)
* Image-text retrieval
* Visual question answering
* Image captioning (with additional decoder, e.g., CoCa)

## Training in OpenCLIP

OpenCLIP supports large-scale training with:

* **Multi-GPU distributed training** - Up to 1024 GPUs tested
* **Multiple data sources** - WebDataset format for billion-scale datasets
* **Efficient batch construction** - Local loss and gradient gathering
* **Mixed precision** - FP16/BF16 for faster training

Example training command:

```bash theme={null}
torchrun --nproc_per_node 4 -m open_clip_train.main \
    --train-data '/data/cc12m/cc12m-train-{0000..2175}.tar' \
    --dataset-type webdataset \
    --batch-size 320 \
    --precision amp \
    --workers 4 \
    --model ViT-B-32
```

## Reference

**Original CLIP Paper:**

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021). *Learning Transferable Visual Models From Natural Language Supervision*. ICML 2021.

📄 [arXiv:2103.00020](https://arxiv.org/abs/2103.00020)

**OpenCLIP Paper:**

Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., ... & Jitsev, J. (2023). *Reproducible scaling laws for contrastive language-image learning*. CVPR 2023.

📄 [arXiv:2212.07143](https://arxiv.org/abs/2212.07143)

## Next Steps

<CardGroup cols={2}>
  <Card title="Contrastive Learning" icon="arrows-split-up-and-left" href="/concepts/contrastive-learning">
    Deep dive into the loss function and training objective
  </Card>

  <Card title="Zero-Shot Classification" icon="wand-magic-sparkles" href="/concepts/zero-shot-classification">
    Learn how CLIP classifies without task-specific training
  </Card>
</CardGroup>